{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/facebookresearch/fairo/blob/master/tutorials/make_new_dataset.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qGoH3yJoTG90"
   },
   "source": [
    "# Datasets for Semantic Parsing Model\n",
    "The semantic parsing model converts plain text to a nested action dictionary that is partially executable. \n",
    "\n",
    "The training dataset for this was created meticulously using a crowdsourcing procedure we describe in detail here: https://www.aclweb.org/anthology/2020.acl-main.427.pdf\n",
    "\n",
    "More details on the existing dataset used for training the semantic parsing model, here : https://colab.research.google.com/github/facebookresearch/fairo/blob/master/tutorials/semantic_parser_onboarding.ipynb\n",
    "\n",
    "\n",
    "\n",
    "## Creating your own dataset\n",
    "If you want to create your own dataset, the high level procedure to follow would be :\n",
    "\n",
    "1. Come up with new natural language command examples.\n",
    "2. Get annotations for these commands.\n",
    "3. Create the training data in the format explained here: \n",
    "https://colab.research.google.com/github/facebookresearch/fairo/blob/master/tutorials/semantic_parser_onboarding.ipynb\n",
    "\n",
    "### Getting annotations for new examples\n",
    "\n",
    "We designed and open-sourced a multi-step web-based tool which asks users a series of multiple-choice questions to determine the semantic content of a sentence. The responses to some questions will prompt other more specific questions, in a process that mirrors the hierarchical structure of the grammar (described here: https://github.com/facebookresearch/fairo/blob/main/base_agent/documents/Action_Dictionary_Spec.md) . \n",
    "\n",
    "These tools are available for use, here: https://github.com/facebookresearch/fairo/tree/main/tools/annotation_tools/text_to_tree_tool .\n",
    "\n",
    "\n",
    "#### **Annotating  the data**\n",
    "\n",
    "If the new commands in your dataset are supported by our grammar (defined here: https://github.com/facebookresearch/fairo/blob/main/base_agent/documents/Action_Dictionary_Spec.md), then you can directly use the tools above with Mechanical Turk to get final annotations.\n",
    "\n",
    "\n",
    "However, if there are commands in your dataset that don't fall under the category of actions (Build, Destroy, Dig, ...etc) or dialogue types (human_give_command, get_memory, put_memory) or are just new additions to what we released, then the existing tool and grammar will need to be extended. \n",
    "\n",
    "The proposed way to go about the extension and a general intuitive explanation about the annotation process is as follows :\n",
    "\n",
    "\n",
    "1. Example command: **\"touch the green chair\"**\n",
    "\n",
    "  1. What kind of dialogue is it ? - Human is giving a command, so `\"dialogue_type\" : \"HUMAN_GIVE_COMMAND\"`\n",
    "  2. Given the dialogue type, other semantic details of the sentence:\n",
    "    - Are there multiple actions ? - Just one single action, so `\"action_sequence\"` list will have one dictionary in it.\n",
    "    - Is there a specific kind of action? Can any of the existing actions be reused ? - We need a new `\"action_type\" : \"TOUCH\"`\n",
    "    - What is it that needs to be touched? `\"reference_object\" : {\"text_span\" : \"the green chair\"}`\n",
    "      - More details about the thing (these go under the \"filters\"):\n",
    "        - `\"has_name\" : \"chair\"` , and `\"has_colour\": \"green\"`\n",
    "    - Does the action need to be repeated ? No.\n",
    "\n",
    "  Based on above, the full dictionary looks like:\n",
    "  ```\n",
    "  { \"dialogue_type\": \"HUMAN_GIVE_COMMAND\",\n",
    "    \"action_sequence\" : [\n",
    "    { \"action_type\" :  \"TOUCH\",\n",
    "      \"reference_object: {\n",
    "          \"filters\" : {\n",
    "            \"has_name\" : [0, [3, 3]],\n",
    "            \"has_colour\": [0, [2, 2]]\n",
    "          }\n",
    "      }\n",
    "    }]\n",
    "  }\n",
    "  ```\n",
    "  where the values for the keys `has_name` and `has_colour` are the indices of the word in the 0th (most recent) sentence in the dialogue.\n",
    "\n",
    "\n",
    "\n",
    "2. Example command: **\"dig three tiny holes there\"**\n",
    "  1. What kind of dialogue is it ? - Human is giving a command, so `\"dialogue_type\" : \"HUMAN_GIVE_COMMAND\"`\n",
    "  2. Given the dialogue type, other semantic details of the sentence:\n",
    "    - Are there multiple actions ? - Just one single action, so `\"action_sequence\"` list will have one dictionary in it.\n",
    "    - Is there a specific kind of action? Can any of the existing actions be reused ? - Yes it : `\"action_type\" : \"DIG\"`\n",
    "    - What is it that needs to be dug? `\"schematic\" : {\"text_span\" : \"three tiny holes\"}`\n",
    "      - More details about the thing that needs to be dug:\n",
    "        - `\"has_name\" : \"holes\"` , and `\"has_size\": \"tiny\"`\n",
    "    - Is there a location where the hole will be dug ? - Yes:\n",
    "      - More details about the location:\n",
    "        - The location is represented using indefinite noun, so `\"location\" : {\"contains_coreference\" : \"yes\"}`\n",
    "    - Does the action need to be repeated ? Yes, three times: `\"repeat\" : {\"repeat_key\" : \"FOR\", \"repeat_count\" : \"three}`\n",
    "\n",
    "  Based on above, the full dictionary looks like:\n",
    "  ```\n",
    "  { \"dialogue_type\": \"HUMAN_GIVE_COMMAND\",\n",
    "    \"action_sequence\" : [\n",
    "    { \"action_type\" :  \"DIG\",\n",
    "      \"schematic: {\n",
    "          \"has_name\" : [0, [3, 3]],\n",
    "          \"has_size\": [0, [2, 2]]\n",
    "          },\n",
    "      \"location\" : {\n",
    "        \"contains_coreference\" : \"yes\" \n",
    "      },\n",
    "      \"repeat\" : {\n",
    "        \"repeat_key\" : \"FOR\",\n",
    "        \"repeat_count\" : [0, [1, 1]]\n",
    "      }\n",
    "    }]\n",
    "  }\n",
    "  ```\n",
    "  where the values for the keys `has_name`, `has_size` and `repeat_count` are the indices of the word in the 0th (most recent) sentence in the dialogue.\n",
    "\n",
    "3. Example command: **\"what colour do you think is that ?\"**\n",
    "  1. What kind of dialogue is it ? - Human is asking a question about something in the environment, so `\"dialogue_type\" : \"GET_MEMORY\"`\n",
    "  2. Given the dialogue type, other semantic details of the sentence:\n",
    "    - What is the question about?  - denoted using the `filters` key in the dictionary. The question is about a reference object in memory, so `\"memory_type\": \"REFERENCE_OBJECT\"`\n",
    "      - More details about the reference object:\n",
    "        - It is represented using indefinite noun (`that`), so `\"contains_coreference\" : \"yes\"`\n",
    "    - Is the question about a specific property of the object or a yes/no question ? - It is asking for the colour, so the expected output is `\"attribute\": \"has_colour\"`\n",
    "\n",
    "  Based on above, the full dictionary looks like:\n",
    "  ```\n",
    "  {\"dialogue_type\": \"GET_MEMORY\", \n",
    "  \"filters\": {\n",
    "    \"memory_type\": \"REFERENCE_OBJECT\", \n",
    "    \"contains_coreference\": \"yes\", \n",
    "    \"output\": {\"attribute\": \"has_colour\"}}}\n",
    "  ```\n",
    "\n",
    "4. Example command: **\"is this a house ?\"**\n",
    "  1. What kind of dialogue is it ? - Human is asking a question about something in the environment, so `\"dialogue_type\" : \"GET_MEMORY\"`\n",
    "  2. Given the dialogue type, other semantic details of the sentence:\n",
    "    - What is the question about?  - denoted using the `filters` key in the dictionary. The question is about a reference object in agent's memory, so `\"memory_type\": \"REFERENCE_OBJECT\"`\n",
    "      - More details about the reference object:\n",
    "        - It has a name : \"house\"  so `\"has_name\": \"house\"`\n",
    "    - Is the question about a specific property of the object or a yes/no question ? - It is a yes/no confirmation like question, so output is returning whether a memory exists: `\"output\": \"memory\"`\n",
    "\n",
    "  Based on above, the full dictionary looks like:\n",
    "  ```\n",
    "  {\"dialogue_type\": \"GET_MEMORY\", \n",
    "  \"filters\": {\n",
    "    \"memory_type\": \"REFERENCE_OBJECT\", \n",
    "    \"has_name\": [0, [3, 3]], \n",
    "    \"contains_coreference\": \"yes\", \n",
    "    \"output\": \"memory\"\n",
    "    }\n",
    "  }\n",
    "  ```\n",
    "  where the value for the key `has_name` is the index of the word \"house\" in the 0th (most recent) sentence in the dialogue.\n",
    "\n",
    "5. Example command: **\"label that very gigantic thing as yellow\"**\n",
    "  1. What kind of dialogue is it ? - Human is giving a label to something in the bot's environment, so `\"dialogue_type\" : \"PUT_MEMORY\"`\n",
    "  2. Given the dialogue type, other semantic details of the sentence:\n",
    "    - What is being tagged?  - denoted using the `filters` key in the dictionary. The label is about a reference object in agent's memory, so `\"memory_type\": \"REFERENCE_OBJECT\"`\n",
    "      - More details about the reference object:\n",
    "        - It has a size : \"very gigantic\"  so `\"has_size\": \"very gigantic\"`\n",
    "        - It is represented using indefinite noun (`that`), so `\"contains_coreference\" : \"yes\"`\n",
    "    - What is the information being given to the agent ? - The colour of the object needs to be tagged as \"yellow\", so update the memory for this as : `\"memory_type\": \"TRIPLE\", \"has_colour\": [0, [7, 7]]`\n",
    "\n",
    "  Based on above, the full dictionary looks like:\n",
    " ```\n",
    " {\"dialogue_type\": \"PUT_MEMORY\", \n",
    "  \"filters\": {\n",
    "    \"has_size\": [0, [2, 3]],  \n",
    "    \"contains_coreference\": \"yes\"}, \n",
    "    \"upsert\": {\n",
    "      \"memory_data\": {\n",
    "        \"memory_type\": \"TRIPLE\", \n",
    "        \"has_colour\": [0, [6, 6]]}}}\n",
    " ```\n",
    " where the values for the keys `has_size` and `has_colour` are the indices of the word in the 0th (most recent) sentence in the dialogue.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YPBMDQm31NOO"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Make new datatset.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
